Distributed discernment system

ABSTRACT

An example of a distributed discernment system including a discernment server and a communications interface permitting bi-directional communications to and from the discernment server; and a plurality of human interface devices, each including a speaker, a microphone, a processor running a local processing program, and a system interface permitting bi-directional communications between the human interface device and the discernment server, where the diagnostic program running on the discernment server is adapted to generate interview instructions provided to the interface devices and the interface devices are adapted receive interview instructions from the discernment server, present a verbal question to a human interviewee; receive and process sensor data from the microphone to determine whether the microphone sensor data corresponds to a complete human voice response to the presented verbal question.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 63/363.494 filed on Apr. 24, 2022.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not applicable.

REFERENCE TO APPENDIX

Not applicable.

BACKGROUND OF THE DISCLOSURE Field of the Invention

The present disclosure relates to apparatus and systems for assessingthe underlying state of a human in a distributed system.

Description of the Related Art

Accurate knowledge of the underlying state of a human being can beimportant for a variety of reasons. For example, knowing whether aparticular human individual is acting with benign or malicious intentcan be critical to the success of an organization of a process. Everyorganization is at risk of being victimized by individuals acting withmalicious intent, such as hidden insiders or individuals seeking toaccess a location or service for improper purposes. For example,government entities are subject to spying attacks where individualsseeking to improperly obtain government, defense, and military secrets.In addition, governmental and industrial organizations are subject totheft, fraud, embezzlement, sabotage and industrial espionage, wheretechnology, trade secrets and other forms of intellectual property areobtained improperly. Still further, locations and services such asstadiums, sporting events and airline travel are attacked by individualsseeking to inflict damages on the location or service (or individual ator associated with the service) through acts of terror or violence.

A significant challenge with addressing the threat posed by maliciousindividuals or groups is that they typically purposefully blend in andgive no overt hints of their malicious intent. They try to hide in plainsight. They are not subject to easy categorization. They could be anyonefrom a senior officer of a company to a newly hired janitor. Manymalicious actors have outstanding reputations, acknowledged forextraordinary performance, and are well-known to organizations they seekto damage. They can be clever and versatile at hiding the nature oftheir threats so that it is impossible to anticipate and protect againstthem, or to take countermeasures before appropriate mitigation processesare fully in place.

Despite the importance of being able to accurately assess the underlyingstate of human beings, prior attempts to do so have been limited both interms of their likelihood of success and/or the challenges required fortheir implementation.

For example, human determination of the underlying state of anotherhuman being is generally inaccurate across a large number of situations.

While technology-focused attempts at deception have been attempted, theyhave not produced acceptable results. For example, one of the mostwidely-known technological approaches—the Polygraph—is generally deemedso inaccurate as to not be admissible in a court of law. While otherapproaches—such as an isolated kiosk-based device—have had more successthan polygraphs, the general costs and logistical challenges associatedwith such systems have generally precluded their widespread adoption.

A further problem with conventional technology-based discernment effortsis that they are not easily scalable, are generally slow, and are noteasily scalable.

It is an object of the disclosed subject matter to overcome thedescribed and other limitations of the prior art.

BRIEF SUMMARY OF THE INVENTION

The present disclosure discloses a distributed system designed forautomated behavioral analysis and credibility assessment of persons todetect hidden deception. The system conducts primary and secondaryscreenings of entrants using a virtual human agent and an array ofnon-invasive sensors to automate the analysis of a person's credibility,documents, and identity.

A brief non-limiting summary of one of the many possible embodiments ofthe present disclosure is a distributed discernment system comprising: adiscernment server including a processer running a diagnostic programand a communications interface permitting bi-directional communicationsto and from the discernment server; a plurality of human interfacedevices, each human interface device including a speaker for presentingaudible stimuli to a human interviewee, a microphone, a processorrunning a local processing program, and a system interface permittingbi-directional communications between the human interface device and thediscernment server; and a communications network permittingbidirectional communications between the discernment server and each ofthe plurality of human interface devices wherein: the diagnostic programrunning on the discernment server is adapted to generate interviewinstructions to be provided to the plurality of human interface devicesover the communications network, wherein the instructions provided to agiven human interface device cause the human interface device to presentaudio stimuli to a human interviewee in the form of a verbal question;each of the plurality of human interface devices is adapted receiveinterview instructions from the discernment server over thecommunications network and the local processing system is configured, inresponse to such instructions, to: present audible stimuli to a humaninterviewee in the form of the verbal question; receive sensor data fromthe microphone following the presentation of the audible stimuli;process the microphone sensor data to determine whether the microphonesensor data corresponds to a complete human voice response to thepresented verbal question; and if the microphone sensor data isdetermined to be a complete human voice response to the presented verbalquestion, provide data to the discernment server indicating that acomplete response to the presented verbal question has been received;and the discernment server is adapted to receive data from each of aplurality of the human interface devices and analyze such data toprovide an assessment of the state of the human interviewee interactingwith each such human interview device.

Additionally or alternatively, each of the plurality of human interfacedevices may further comprises an eye tracker, and wherein the microphonegenerates a timestamped audio data stream; the eye tracker generates atimestamped eye tracking data stream, and wherein each of the pluralityof human interface devices further comprises: means for streaming thetimestamped eye tracking data stream and the timestamped audio datastream to the discernment server in such a manner that the data streamsare associated with an interview ID; means for generating timestampsassociated with the start and stop points of the presentation of theverbal question to a human interviewee interacting with such device;means for generating timestamps associated with the start and stoppoints of the human interviewee's response to the presented verbalquestion; and means for transmitting to the discernment servertimestamped data associated with the start and stop points of thepresented verbal question and timestamped data associated with the startand stop points of the human interviewee's response to the presentedverbal question.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The following figures form part of the present specification and areincluded to demonstrate further certain aspects of the presentinvention. The invention may be better understood by reference to one ormore of these figures in combination with the detailed description ofspecific embodiments presented herein.

FIG. 1 illustrates an exemplary embodiment of a distributed discernmentsystem 1000 constructed in accordance with certain teachings of thisdisclosure.

FIG. 2A illustrates a Manage Questions administrative portal that can beused by an administrator to manage various questions that can be used ininterviews.

FIG. 2B illustrates an exemplary Manage Projects portal.

FIG. 2C illustrates a Manage Interview portal that can be used toconfigure interviews.

FIGS. 3A-3H illustrates an exemplary embodiment of a local appliance2000.

FIGS. 4A-4C illustrate various components and elements positioned withinthe interior space defined by the chassis 2100 and the back cover 2900.

FIG. 5 illustrates aspects of an exemplary process that permits timesynchronization of the provided stimuli and the received sensor data.

FIG. 6 illustrates an example of the type of data that may be providedfor the questions and answers.

FIG. 7B shows one process that may be implemented by an end of speechdetector to determine whether an end of speech event has occurred.

FIG. 7C illustrates an exemplary embodiment wherein both audio data andvisual data are processed to detect an end of answer/end of speechcondition and/or a no response condition.

FIG. 8A illustrates an exemplary report interface.

FIG. 8B illustrates an example form of an interview report.

While the inventions disclosed herein are susceptible to variousmodifications and alternative forms, only a few specific embodimentshave been shown by way of example in the drawings and are described indetail below. The figures and detailed descriptions of these specificembodiments are not intended to limit the breadth or scope of theinventive concepts or the appended claims in any manner. Rather, thefigures and detailed written descriptions are provided to illustrate theinventive concepts to a person of ordinary skill in the art and toenable such person to make and use the inventive concepts.

DETAILED DESCRIPTION

The Figures described above, and the written description of specificstructures and functions below, are not presented to limit the scope ofwhat I have invented or the scope of the appended claims. Rather, theFigures and written description are provided to teach any person skilledin the art to make and use the inventions for which patent protection issought. Those skilled in the art will appreciate that not all featuresof a commercial embodiment of the inventions are described or shown forthe sake of clarity and understanding. Persons of skill in this art willalso appreciate that the development of an actual commercial embodimentincorporating aspects of the present inventions will require numerousimplementation-specific decisions to achieve the developer's ultimategoal for the commercial embodiment. Such implementation-specificdecisions may include, and likely are not limited to, compliance withsystem-related, business-related, government-related, and otherconstraints, which may vary by specific implementation, location andfrom time to time. While a developer's efforts might be complex andtime-consuming in an absolute sense, such efforts would be,nevertheless, a routine undertaking for those of skill in this arthaving benefit of this disclosure. It must be understood that theinventions disclosed and taught herein are susceptible to numerous andvarious modifications and alternative forms. Lastly, the use of asingular term, such as, but not limited to, “a,” is not intended aslimiting of the number of items. Also, the use of relational terms, suchas, but not limited to, “top,” “bottom,” “left,” “right,” “upper,”“lower,” “down,” “up,” “side,” and the like are used in the writtendescription for clarity in specific reference to the Figures and are notintended to limit the scope of the invention or the appended claims.

Aspects of the inventions disclosed herein may be embodied as anapparatus, system, method, or computer program product. Accordingly,specific embodiments may take the form of an entirely hardwareembodiment, an entirely software embodiment or an embodiment combiningsoftware and hardware aspects, such as a “circuit,” “module” or“system.” Furthermore, embodiments of the present inventions may takethe form of a computer program product embodied in one or more computerreadable storage media having computer readable program code.

Items, components, functions, or structures in this disclosure may bedescribed or labeled as a “module” or “modules.” For example, but notlimitation, a module may be configured as a hardware circuit comprisingcustom VLSI circuits or gate arrays, off-the-shelf semiconductors suchas logic chips, transistors, or other discrete components. A module alsomay be implemented as programmable hardware devices such as fieldprogrammable gate arrays, programmable array logic, programmable logicdevices, or the like. Modules also may be configured as software forexecution by various types of processors. A module of executable codemay comprise one or more physical or logical blocks of computerinstructions that may be organized as an object, procedure, or function.The executables of a module need not be physically located together butmay comprise disparate instructions stored in different locations thatwhen joined logically together, comprise the module and achieve thestated purpose or function. A module of executable code may be a singleinstruction, or many instructions, and may even be distributed overseveral different code segments, among different programs, and acrossseveral memory devices. Similarly, data may be identified andillustrated herein within modules, and may be embodied in any suitableform and organized within any suitable type of data structure. The datamay be collected as a single dataset, or may be distributed overdifferent locations including over different storage devices, and mayexist, at least partially, merely as electronic signals on a system ornetwork. Where a module or portions of a module are implemented insoftware, the software portions may be stored on one or more computerreadable storage media.

When implementing one or more of the inventions disclosed herein, anycombination of one or more computer readable storage media may be used.A computer readable storage medium may be, for example, but notlimitation, an electronic, magnetic, optical, electromagnetic, infrared,or semiconductor system, apparatus, or device, or any suitablecombination of the foregoing. More specific, but non-limiting, examplesof the computer readable storage medium may include the following: aportable computer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a portable compact disc read-only memory (CD-ROM), adigital versatile disc (DVD), a Blu-ray disc, an optical storage device,a magnetic tape, a Bernoulli drive, a magnetic disk, a magnetic storagedevice, a punch card, integrated circuits, other digital processingapparatus memory devices, or any suitable combination of the foregoing,but would not include propagating signals. In the context of thisdisclosure, a computer readable storage medium may be any tangiblemedium that can contain or store a program for use by or in connectionwith an instruction execution system, apparatus, or device.

Computer program code for carrying out operations of one or more of thepresent inventions may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Python, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. The remotecomputer may be connected to the user's computer through any type ofnetwork, including a local area network (LAN) or a wide area network(WAN), or the connection may be made to an exterior computer forexample, through the Internet using an Internet Service Provider.

Reference throughout this disclosure to “one embodiment,” “anembodiment,” or similar language means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least one of the many possible embodiments of thepresent inventions. The terms “including,” “comprising,” “having,” andvariations thereof mean “including but not limited to” unless expresslyspecified otherwise. An enumerated listing of items does not imply thatany or all of the items are mutually exclusive and/or mutuallyinclusive, unless expressly specified otherwise. The terms “a,” “an,”and “the” also refer to “one or more” unless expressly specifiedotherwise.

Furthermore, the described features, structures, or characteristics ofone embodiment may be combined in any suitable manner in one or moreother embodiments. In the following description, numerous specificdetails are provided, such as examples of programming, software modules,user selections, network transactions, database queries, databasestructures, hardware modules, hardware circuits, hardware chips, etc.,to provide a thorough understanding of embodiments of the disclosure.Those of skill in the art having the benefit of this disclosure willunderstand that the inventions may be practiced without one or more ofthe specific details, or with other methods, components, materials, andso forth. In other instances, well-known structures, materials, oroperations are not shown or described in detail to avoid obscuringaspects of the disclosure.

Aspects of the present disclosure are described below with reference toschematic flowchart diagrams and/or schematic block diagrams of methods,apparatuses, systems, and computer program products according toembodiments of the disclosure. It will be understood by those of skillin the art that each block of the schematic flowchart diagrams and/orschematic block diagrams, and combinations of blocks in the schematicflowchart diagrams and/or schematic block diagrams, may be implementedby computer program instructions. Such computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus tocreate a machine or device, such that the instructions, which executevia the processor of the computer or other programmable data processingapparatus, structurally configured to implement the functions/actsspecified in the schematic flowchart diagrams and/or schematic blockdiagrams block or blocks. These computer program instructions also maybe stored in a computer readable storage medium that can direct acomputer, other programmable data processing apparatus, or other devicesto function in a particular manner, such that the instructions stored inthe computer readable storage medium produce an article of manufactureincluding instructions which implement the function/act specified in theschematic flowchart diagrams and/or schematic block diagrams block orblocks. The computer program instructions also may be loaded onto acomputer, other programmable data processing apparatus, or other devicesto cause a series of operational steps to be performed on the computer,other programmable apparatus or other devices to produce a computerimplemented process such that the instructions that execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The schematic flowchart diagrams and/or schematic block diagrams in theFigures illustrate the architecture, functionality, and/or operation ofpossible apparatuses, systems, methods, and computer program productsaccording to various embodiments of the present inventions. In thisregard, each block in the schematic flowchart diagrams and/or schematicblock diagrams may represent a module, segment, or portion of code,which comprises one or more executable instructions for implementing thespecified logical function(s).

It also should be noted that, in some possible embodiments, thefunctions noted in the block may occur out of the order noted in thefigures. For example, two blocks shown in succession may, in fact, beexecuted substantially concurrently, or the blocks may sometimes beexecuted in the reverse order, depending upon the functionalityinvolved. Other steps and methods may be conceived that are equivalentin function, logic, or effect to one or more blocks, or portionsthereof, of the illustrated figures.

Although various arrow types and line types may be employed in theflowchart and/or block diagrams, they do not limit the scope of thecorresponding embodiments. Indeed, some arrows or other connectors maybe used to indicate only the logical flow of the depicted embodiment.For example, but not limitation, an arrow may indicate a waiting ormonitoring period of unspecified duration between enumerated steps ofthe depicted embodiment. It will also be noted that each block of theblock diagrams and/or flowchart diagrams, and combinations of blocks inthe block diagrams and/or flowchart diagrams, may be implemented byspecial purpose hardware-based systems that perform the specifiedfunctions or acts, or combinations of special purpose hardware andcomputer instructions.

The description of elements in each Figure may refer to elements ofproceeding Figures. Like numbers refer to like elements in all figures,including alternate embodiments of like elements. In some possibleembodiments, the functions/actions/structures noted in the figures mayoccur out of the order noted in the block diagrams and/or operationalillustrations. For example, two operations shown as occurring insuccession, in fact, may be executed substantially concurrently or theoperations may be executed in the reverse order, depending upon thefunctionality/acts/structure involved.

The Overall Distributed System: FIG. 1 illustrates an exemplaryembodiment of a distributed discernment system 1000 constructed inaccordance with certain teachings of this disclosure. As illustrated inthe figure, in this general embodiment, the distributed discernmentsystem comprises four main components: (a) one or more cloud-baseddiscernment server systems 1100; (b) a plurality of human interfacedevices 1200; (c) a plurality of administrative access devices 1300 and(d) one or more communication networks 1400, 1450 each permittingbi-directional communication between the cloud-based discernment serversystem (or systems) 1100 and the human interface devices 1200, fornetwork 1400, and/or between the cloud-based discernment server system1100 and the administrative access devices 1300, for network 1450.

The communication networks 1400, 1450 may comprise any suitable networkfor enabling bi-directional communications and may take the form ofwired networks, wired networks, and networks comprising wired andwireless links. In exemplary embodiments, the communication networks1400 and/or 1450 may include general Internet connections supported byexisting infrastructure including Wi-Fi routers, Ethernet connections,fiber optic connections and any other suitable connection. Thecommunication networks 1400 and/or 1450 may further include cellularnetworks, such as 4G or 5G networks.

In the exemplary embodiment, each of the human interface systems is usedto support one or more interactions with a specific human. For purposesof this disclosure, a discrete grouping of interactions between thedistributed discernment system and a human individual is referred to asan “interview” and the human individual involved in a given interview isreferred to as an “interviewee.”

It should be understood that an interview may consist of one or a moredifferent interactions between the discernment system and a giveninterviewee, and that the interactions may take the form of theprovision of various different stimuli to the interviewee and thedetection of the response (or responses to those stimuli). For example,in one form, an interview could involve interactions where questions arevisually presented to an interviewee and the interviewee is requested torespond by typing answers on a keyboard. In other embodiments, theinterview could take the form of an interactive interface that utilizesa virtual person to conduct a verbal question-and-answer interrogationof the interviewee, where questions are posed verbally, and theinterviewee is requested to articulate their response. In still otherexamples, the interactions could take the form of the presentations ofimages, sounds, smells, or the like to the interviewee and thedetermination of the interviewee's responses to those stimuli. Stillfurther, the interactions comprising an interview could take the form ofa combination of the above-described and other forms of stimuli.

In addition to presenting the communications giving rise to theinterview to the interviewee, the human interface system will alsodetect certain reactions of the interviewee to the stimuli. For example,the human interface system can include microphones to capture theinterviewee's audible response to presented questions. It can alsoinclude a camera and an eye tracker for detecting the interviewee'sposture and eye gaze during the interview. It could also include avariety of other detectors and sensors for detecting other responsesfrom the interviewee, such as posture changes, pulse rate changes,changes in skin activity (e.g., pore opening, sweating, temperaturechanges, etc.). As described in more detail below, the human interfacesystem(s) will transmit data reflecting the sensed and detectedattributes of the interviewee to one or more discernment server systems.

In the illustrated embodiment, the discernment server system(s)interacts with the human interface systems (through the communicationsystem(s)) in such a manner that the discernment server systemdetermines a variety of desired interactions for a given interview.These interactions can either be scripted—in the sense that for acertain interviewees or groups of interviewees—the same series ofquestions can always be presented in the same order or dynamic. In adynamic interview, for a given interviewee, the discernment serversystem(s) will request the human interface system to establish certaininteractions with the interviewee and will then use the responsesreceived from certain initial interactions to determine which (if any)subsequent interactions to request.

At a high level the operation of this illustrated embodiment is asfollows.

At a first time a system user (referred to herein as a Tenant or anAdministrator) will configure the to define at least one interview. Ingeneral, each interview will be defined by a series of stimuli to bepresented to the interviewee during the interview. In one example, thisis done through the Administrator accessing a configuration portalthrough use of one of the administrative access devices 1300.

The, at a later time, the stimuli comprising the defined interview willbe presented to the interviewee by the specific human interface system1200 and the interviewee will interact with the human interface systemin response to the one or more initial interactions. The response (orresponses) received by the human interface system 1200 in response tothe interview stimuli will then be transmitted from the human interfacesystem 1200 to the cloud-based discernment server system 1100 (eitherwith or without some local processing). The cloud-based discernmentserver system 1100 will then receive and process the receivedresponse(s) and, in response, may generate a subsequent set of requestedinteractions to be transmitted to the human interface system. The humaninterface system can then present the subsequent interactions to theinterviewee, and receive responses from the interviewee. The process maybe repeated a number of times with a number of subsequent interactionspresented to the interviewee and a number of subsequent responsesreceived by the discernment server system.

In the above example, once the interview is completed the cloud-baseddiscernment server system 1100 can then process the received response toprovide a general assessment of the underlying state of the interviewee.For example, in applications where the system is utilized to determinethe extent to which the interviewee is likely to be engaged in maliciousor unauthorized activity with respect to an employer, the discernmentserver system may generate an indication of the general assessedpotential that the individual is (or is likely to engage in) maliciousor unauthorized activity (e.g., deception detected, medium risk, lowrisk, no anomalies detected). As another example, in applications wherethe system is used for access control at a sporting event, the systemmay either automatically open a gate to permit an individual to passthrough into the event or actuate a mechanism to direct the intervieweeto an area dedicated to secondary or more through screening.

Once the interview data is processed, the cloud-based discernment systemcan then generate one or more notice or reports associated with theinterview that can be made available to authorized Administratorsthrough one or more of the administrative access devices 1300.

The embodiment depicted in FIG. 1 allows the activities described aboveto be performed concurrently in different locations and with respect todifferent administrators, interviews, and interviewees. Thus, forexample, one administrator—associated, for example with an airline—mayconfigure interviews that are used to screen multiple airline passengersfor security risks while, at the same time, another administrator—forexample, one interviewing employees for insider threat detection—may beconducting interviews of its employees to screen for embezzlement ortheft of trade secret activities.

In the exemplary embodiment, customer interface apparatuses 1300 arealso provided to allow authorized persons and entities to access thediscernment server system for modifying or adjusting the discernmentserver system, receiving reports concerning the operation of the system,receiving analysis and/or reports concerning one or multiple humaninteractions conducted by the system, or for any other purpose.

By judiciously determining where and how various aspects of thedescribed process are implemented, the embodiments of the present systemprovide a highly-flexible, highly-scalable, cost-effective and robustsystem for discerning the underlying state of humans suitable for alarge number of applications.

Various aspects, and several of the many possible alternativeembodiments of the exemplary distributed discernment system will beexemplified below. When considering the following written description itwill be understood by those of skill in the art that the variousembodiments are non-limiting and structural components and/or functionalcharacteristics may be combined, a la carte style, to provide systemshaving various structural configurations and functionality. For example,and without limitation, as discussed in more detail below, each of thehuman interface systems in a particular embodiment of the distributeddiscernment system may take the form of any of a stationary system, amobile system, a desktop system, a tablet-based system, or a smartphonesystem, and other interface systems that may be envisioned by those ofordinary skill in the art. The discussion of an embodiment utilizingdesktops is in no way intended to preclude a system that would combinehuman interface systems having other forms such as a desktop form, atablet form, and/or smartphone forms. Those ordinarily skilled in theart may practice the inventions taught and disclosed herein with theseand many other forms and combinations. Accordingly, unless explicitlynoted otherwise, all exemplary embodiments and all exemplary variantembodiments disclosed herein should be understood to be combinable withall other envisioned embodiments and variants to achieve the statedpurposes and results of the inventions described herein.

The Human Interface System: As generally described above, each humaninterface system of the present disclosure is a system that permits theoverall system to interface with one or more human interviewees to both:(a) present stimuli to a human interviewee and (b) receive and detectattributes of a human interviewee, including specifically responses froma human interviewee to provided stimuli.

Stimuli and Output Apparatus: The stimuli provided to each humaninterviewee, and the apparatus within each human interface systemproviding such stimuli, can vary depending on the application of theoverall system. In a most basic case, the stimuli can consist solely ofaudible stimuli in the form of questions presented to the humaninterviewee. In such embodiments, the human interface system maynecessarily include one or more audio speakers for providing the audiblemessages.

In more typical embodiments the stimuli provided by the human interfacesystem may include audible stimuli (described above) and visual stimuli.As with the audible stimuli, the visual stimuli may take various formsincluding but not limited to words, static images, video clips,displayed 2D or 3D images, displayed physical objects, a virtual humanagent (which could take the form of a 2D or 3D moving image, or acartoon-like humanoid character), or any other suitable form.

Detected Attributes and Sensors: The specific interviewee attributesdetected by the human interface system will vary depending on theapplication, the nature of the stimuli provided for a specificembodiment, and other factors, such as cost, size and bandwidthconstraints that may be placed on the system. In many preferredembodiments, the detected attributes (and their associated detectingsensors) will be attributes that can be detected non-invasively (i.e.,without making physical contact with the human interviewee). Suchattributes include, for example, verbal responses, eye movement, generalbody posture, facial expressions, etc.)

In a basic case, the human interface system may include detectors fordetecting audible/verbal responses from an interviewee, visualinformation concerning visible aspects of the interviewee and eyemovement.

To detect audible/verbal responses a microphone (or microphone array)may be utilized. The received audible data can be analyzed to determinevocalic aspects of the interviewee's responses, such as pitch,pitch-changes, rate of speech, tempo, volume/intensity etc. The receivedaudible data can also be processed to provide linguistic data related tothe interviewee's response such as the specific informational content ofthe verbal response (i.e., what is being said such as “yes”, “uh-hu”, “Idon't know”, to much more complicated responses); the extent of pronounusages as opposed to more specific references such as hedging,avoidance, etc.

To detect visual aspects of the interviewee, one or more cameras may beemployed. To detect eye movement, one or more eyer trackers may beutilized. The eye trackers used in the disclosed system may take one ofmany forms. In certain examples, the eye trackers may be dedicatedapparatus built into a specific device. Such dedicated eye trackers mayinclude, for examples, eye trackers available from Tobii, Gazepoint,ISCAN or others.

The Interview and the Scheduling of Interviews: In one exemplaryembodiment each human interface device will be a device that is capableof receiving from the cloud base discernment server 1100 data associatedwith an interview, collecting an interviewee's response to the stimuliassociated with the interview and then transmitting the received data(after some processing) the cloud-based discernment server 1100 forprocessing.

In the example discussed herein, each interview is associated with aseries of questions, with each question being associated with bothaudible stimuli to be presented to the user and some visual stimulationto be presented to the user with the audible stimuli. In the specificexample discussed below, each question will take the form of a verbalquestion to the interviewee and an image that can take the form of anindividual speaking to ask the verbal question and/or an imageassociated with the question. Thus for example, one question can takethe form of 3D video image of a human attendant asking the interviewee“Did you pack your bags yourself?” As another example, another questioncan take the form of an image of a page from of an interviewee'spassport associated with the question “Are you the person whose pictureappears in this Passport?”. It will be appreciated that the questionsdiscussed above are exemplary only and that various other forms ofquestions can be implemented without departing from the scope of thepresent disclosure, including questions consisting of only images orsounds to which a human response will be detected.

In one embodiment of the system disclosed herein, the administrativeaccess devices 1300, in conjunction with the cloud-based discernmentserver 1100 or another suitable processing system can be used to permitmultiple administrators to define and manage a number of differentquestions. An exemplary portal that can be used for such activities isshown in FIG. 2A.

FIG. 2A illustrates a Manage Questions administrative portal that can beused by an administrator to manage various questions that can be used ininterviews.

As depicted in FIG. 2A, the Manage Questions portal permits anadministrator (or Tenant) to add questions to a list of availablequestions and modify/change adjust the data associated with existingquestions. As reflected in FIG. 2A, in the exemplary system, the ManageQuestion portal permits an administrator to assign a question name,question type, issue type and action to a variety of questions. Thisportal can then be used to develop and maintain a library of questionsthat the administrator can use to assembly one or more interviews.

In the illustrated example, the questions available to an administratorthrough the Manage Questions portal can include two types of questions.First, questions that will be available to all, or multipleadministrators using the distributed discernment system 1000, includingadministrators associated with different organizations and groups. Suchquestions can include questions common to different types of interviews,such a, for examples: “Please state your name?”; “Are you the personwhose name appears on this document?”; “Are you currently under theinfluence of alcohol or any non-prescribed drug or medication?”. Secondthe Manage Questions portal can be used to create and maintain questionsspecific to—and accessible only to—one administrator or a particularadministrator. Such questions can include questions of interest only toa particular administrator or group of administrators, such as—forexample—“Have you ever intentionally left the West-Side gate open at theend of a shift?”; “Have you ever entered Office 3B on the Third Floor?”;“Do you recognize this individual?,” “Are you currently seekingemployment outside the Company?”.

As reflected in FIG. 2A, in the illustrated example, each question canbe configured to be a question type that is associated with an image orwithout an image. Each question can also be associated with a particularissue tyle such as, for example: “Drug Test”; “Security Test”;“Retention Test”.

In the exemplary system disclosed herein, the administrative accessdevices 1300 can also be used by administrators to create and manageprojects, where each project can be associated with a given interview,multiple interviews, or a given series of interviews. Each such projectcan be configured through a Manage Projects portal through which anadministrator can define a project by name and associate various issueswith that project. FIG. 2B illustrates an exemplary Manage Projectsportal. As shown in the figure, each project can be assigned a projectname, can be associated with different issues, and a description. Asfurther reflected in FIG. 2B, a number of interviews can be scheduledfor each project.

Although not reflected in FIG. 2B, each project can be associated withone or more specific interviews, with each interview associated with aspecific sequence of questions selected from those questions within theManage Questions library. In this manner, an administrator can configurethe system to implement mule projects, or different administrators canconfigure the system to implement multiple projects, with each projectbeing associated with a particular set of interviews and one or moreparticular issues.

In the exemplary system, the various questions can be used to create oneor more scheduled interviews, which can take place within a project.FIG. 2C illustrates a Manage Interview portal that can be used toconfigure interviews. As shown in the figure, each such scheduledinterview can be associated with an interview name, the specific issuesto be addressed in the interview, the date the interview (or interviewsare to be conducted), and the specific number of scheduled interviews tobe conducted on the specified date.

Although not illustrated in FIG. 2C, each scheduled interview can be—nutneed not be—associated with a particular interviewee. Thus for example,a project is intended to interview all employees within a particulargroup in an organization, each scheduled interview may be associatedwith a particular employee. In other applications, such as airportscreening, there may be a desire to have the interviews conducted on ananonymous basis (or on a basis where the interviewee's name is not knownbefore the interview begins). For such applications, there will be noassociation—and at least no prior association—of the interviews and anyinterviewees.

It should be noted that the administrative access devices 1300 need notnecessarily share data with the cloud-based discernment system 1100 forpurposes of question creation, project management or interviewmanagement. Thus, for example, each different administrator or grouputilizing the exemplary system can have their own administrative accessdevices that are isolated from and securely separated from theadministrative access devices used by other administrators. Thus, forexample, a first administrator associated with one group, such as anairline, can define questions, associated them into interviews, andcreate projects associated with such interviews in such a manner thatthe questions, interviews and projects are not accessible to any otheradministrator. As another example, an administrator who wants to arrangeinterviews of a group of specific employees, can arrange for interviewsof those employees without any other administrator—or even thecloud-based discernment server—knowing what specific interviews will beconducted. All that the cloud-based discernment server will know, forexample, is that interviews have been scheduled, that they will eachinclude a defined number of questions.

In one exemplary embodiment, administrators desiring a high degree ofsecurity may be able to maintain their own separate, private and secureadministrative access devices that can be used to define questions,interviews, projects and/or to manage interviews. In such embodiments,the cloud-based server will only have access to limited informationconcerning the interviews to be conducted, such as how many questionswill be asked and data sufficient for the cloud-based server to ensurethat the data associated with each question is provided to the humaninterface device 1200 conducting the interview and to ensure that thedetected response to each question is provided to the cloud-based seversystem 1100. The cloud based server-system—in such an example—canprocess the received data for each question and provide an indication tothe specific administrative access device 13000 as to whether likelydeception was detected with respect to each question. In this example,while the administrator with access to the administrative access deviceat issue may have knowledge of the nature of the specific question, andthe specific identity of the individual interviewee who responded to thequestion, the cloud-based discernment system 1100 would not have suchdata. It would have conducted its analysis anonymously.

As the above makes clear, during operation of the exemplary systemdescribed herein for a specific interview, data associated with specificquestions will be provided to a human interface device 1200 conductingthe interview, which will present the questions to an interview, detectthe responses and forward data associated with the responses to thecloud-based discernment system 1100. In one exemplary embodiment thedata associated with each question (e.g., the data associated with theaudio, visual and/or other stimuli) will be provided to the humaninterview device 1200 on a per question basis as the interview proceeds.In another exemplary embodiment, the data associated with multiplequestions (and potentially all questions) will be provided to the humaninterview device 1200 in advance of each interview. In still otherapplications, the cloud-based discernment system 1100 may not providethe data associated with the various questions to the human interfacedevice 1200, but may rather interact with one or more of theadministrative access devices 1300 such that the administrative accessdevice 1300 or devices 1300 provide the data associated with eachquestion to the human interface devices 1200. In such applications, thecloud-based discernment server 1100 will receive data indicating that aquestion has been asked and data associated with the interviewee'sresponses. The cloud-based discernment server 1100 can then process theresponses to provide an indication of whether they likely reflect adeceptive response.

The Physical Human Interface Device: In one embodiment, one or more ofthe human interface devices 1200 may take the form of a generally mobileintegrated local appliance where the generally mobile integratedappliance may have a form factor somewhat like a tablet computer. FIGS.3A-3H illustrates an exemplary embodiment of a local appliance 2000.

In the example of FIGS. 3A-3H, the local appliance 2000 is intendedprimarily to perform the functions of: (i) initiating an interview witheach interviewee; (ii) presenting notices, data, information and stimulito each interviewee for purposes of the interview; (iii) capturing dataassociated with each interview (e.g., received audio data; receivedvideo data; eye-tracking data; timing, etc.); and (iv) providingcaptured data to the cloud-based discernment system 1100 for processingby such system.

FIGS. 3A-3C provide a front external view of the local appliance 2000.

As best reflected in the FIG. 3A, the local appliance 2000 includes afront chassis element 2100 that may be formed, for example of metal or ametal blend such as a magnesium/aluminum blend. The front chassiselement 2100 defines a first, generally flat section 2110; a secondsection 2120 that, in the illustrated example extends outwardly from thefirst generally flat section at an angle such that the upper surface ofthe second section will generally face upwards towards a user of theappliance; and a third section 2130 that generally faces forward fromthe local appliance 2100.

In the illustrated example, when the local appliance 2000 is positionedon a flat surface: (i) the first section 2110 is generally angled suchthat its exterior surface will be generally angled up towards aninterviewee positioned in front of the appliance 2000; (ii) the secondsection 2120 is generally angled such that its exterior surface will begenerally angled up towards an interviewee positioned in front of theappliance 2000 where the angle defined by the second section 2120 withrespect to the horizontal is greater than the angle defined by the firstsection 2110 with respect to the horizontal; and (iii) the third section2130 is generally angled downwards with respect to an intervieweepositioned in front of the appliance 2000.

As discussed in more detail below, infrared illuminators and an eyetracker are associated with the second section 2120. It has beendiscovered that having the second section 2120 angled within a range of10 (ten) to 20 (twenty) degrees with respect to the horizontal, and morespecifically 15 (fifteen) degrees with respect to the horizontal isbeneficial in that it allows the eye tracker to maintain appropriatetracking of an interviewee's eyes over a broad range of interviewee headand body movement.

While the above description used the term “interviewee” to refer to aperson positioned in front of the local appliance, it will be understandthat the term is intended to encompass any user of the appliance, suchthat it should be understood to include both those interacting with theappliance 2000 during the course of a formal interview and thoseinteracting with the appliance as an administrator configuring theappliance, a technician configuring or repairing the local appliance, anindividual being interviewed by the appliance, or any other person whointerfaces or interacts with the appliance in any way.

As shown in FIGS. 3A and 3B a display screen 2200 for presenting videostimuli and interface information to a user is generally positionedwithin or on the first, generally, flat section of the chassis element2100. The display screen 2200 may take the form of a generallyflat-panel LCD display element.

A protective cover and/or bezel (not separately labeled) may extend overthe front portion of the display screen and the front cover may extendover other elements and components of the appliance 2000. FIG. 3B-1illustrates the embodiment of FIG. 3A with such covers/bezel's removedand FIG. 3B-2 illustrates the embodiment of FIG. 3B-1 with a speakermesh element (discussed below) removed.

As FIGS. 3A, 3B-1 and 3B-2 reflect, the depicted exemplary localappliance 2000, various input and/or output devices and sensor orsensor-related elements are positioned around the display screen 2200.As shown in FIGS. 3B-1, 3B-2 , and, in some respects, also in FIG. 3A,these devices, sensors, and sensor-related elements include an ambientlight and/or proximity sensor 2210; a digital camera 2220 for capturingthe facial image of an interviewee (hereinafter referred to as thecamera or face camera); and multi-element digital microphone 2310, 2320located along the top portion of the chassis element 2100 above thedisplay screen 2200. In the illustrated example, the multi-elementdigital microphone is depicted as being formed from two elements 2310,2310 but it will be understood that the multi-element digital microphonecan include fewer or more elements. In one exemplary embodiment a thirdmicrophone element (not shown in FIGS. 3A, 3B-1 or 3B-2 , but that couldbe positioned on the rear side of the appliance 2000) could be used to,for example, filter out background noise and permit betteridentification of an interviewee's vocalic response.

In the illustrated example, the digital camera or face camera 2220 islocated approximately at the midpoint of the width of the localappliance 2000, above the display screen 2200 and is designed to capturevisible images. In the example, the ambient light and/or proximitysensor 2210 is located on one side of the camera at a location that isin substantially the same horizontal plane as the camera 2220 and thatis located closer to the camera 2000 than to the side edge of theappliance. In the example, of FIGS. 3A and 3B-1 and 3B-2 , the ambientlight and/or proximity sensor 2210 is located within two (2) inches ofthe camera. In the illustrated example, two of the elements of themulti-element digital microphone 2310, 2320 are located on the side ofthe camera 2220 opposite the ambient light sensor and/or proximitysensor.

As shown on the figures, a first one of the digital microphone elements2310 is located closer to the camera 2220 than to the side edge of theappliance, while a second one of the digital microphone elements islocated closer to the side edge of the appliance than to the camera2220. In the example of FIGS. 3A and 3B-1 and 3B-2 , the first digitalmicrophone element 2310 is located within two inches of the camera 2220and the second digital microphone element 2320 is located within twoinches of the side edge of the local appliance. As noted, both digitalmicrophone elements 2310, 2320 are in substantially the same horizontalplane as the camera 2220 and the ambient light and/or proximity sensor.

As best shown in FIGS. 3B-1 and 3B-2 s, first and second infra-redilluminators 2410 and 2420 and an eye tracking sensor 2500 are locatedon the second, generally upward facing section 2120 of the chassis 2100.In the illustrated example, the eye tracking sensor 2500 is locatedsubstantially at a midpoint across the width of the appliance 2000 andeach of the first and second infra-red illuminators 2410, 2420 arelocated along substantially the same horizontal plane as the eyetracking sensor 2500 and at locations that are closer to a side edge ofthe appliance than to the eye tracking sensor 2500. In the example ofFIGS. 3B-1 and 3B-2 , each infrared illuminator 2410, 2420 is locatedwithin two inches of a side edge of the appliance.

As further shown in the figures, and best reflected in FIG. 3B-2 , firstand second speaker elements 2610, 2620 are positioned on a plane belowthe second upward facing section of the chassis. As shown in thefigures, in the illustrated embodiment, the plane in which the speakersare positioned extends downwardly from the second upward facing surface2120 such that, when the appliance is resting on a surface, they will bedirected slightly downward from the appliance 2000 towards the surfaceon which the appliance rests. As reflected in the figures, each of thetwo speakers 2610, 2620 is located closer to a side edge of theappliance than to the center of the appliance, such that each speakerincludes a section that is within two inches of a side edge of theappliance.

In the exemplary embodiment, as best shown in FIG. 3B-2 , a lightinteractive element 2700 is provided that takes the form of alight-emitting-diode assembly (and associated printed circuit board) ispositioned between the first and second speaker elements 2610, 2620substantially at the center of the appliance. As discussed in moredetail below, the light interactive element 2700 can be used to providevisible cues to a user of the appliance 2000. In one exemplaryembodiment, the light interactive element comprises a 3×12 LED array.

In one exemplary embodiment of the appliance depicted in example ofFIGS. 3A and 3B-1 and 3B-2 , the first and second speaker elements 2610,2620 and the light interactive element 2700 are positioned behind a meshelement to which speaker fabric is attached. Details of such anembodiment are shown in FIG. 3C and 3D.

FIG. 3C illustrates details of the front chassis 2100 within the displayscreen removed. As shown in the figure, the front chassis 2100 is formedsuch that it includes a recessed area 2102 for receiving the displayscreen and openings 2104, 2106 for enabling connections to and airflowto and from the rear side of the display screen. The chassis 2100 alsodefines openings and/or recesses 2108, 2110 for coupling to and/orpassage of light through and from camera and ambient light sensingand/or proximately sensing elements 2220, 2100. In the example of FIG.2C, the chassis 2100 further defines elements 2312, 2314 for receipt offirst and second digital microphone elements 2310, 2320.

As further shown in FIG. 2C, the exemplary chassis 2100 further definesopenings 2412, 2414 permitting attachment of and/or communication with,first and second infra-red illuminators (described above) and an opening2502 permitting access to an eye tracking sensor.

As still further shown in FIG. 2C, the exemplary chassis 2100 definesopenings 2612, 2622 for attachment and/or communication with first andsecond speakers and an opening 2702 permitting access to a lightinteractive element. As shown in FIG. 3C a speaker mesh element 2800 ispositioned across the front of the chassis 2100 to cover the first andsecond speaker elements and the light interactive element. As depicted,the speaker mesh element 2800 includes groupings of a plurality ofopenings 2802, 2804 to permit the passage of sound therethrough.

The speaker mesh element 2800 may be formed from a frosted polycarbonateand can perform two functions. First, the speaker mesh element 2800 mayprovide a base to which a speaker fabric material may be attached toboth block passage of material from the exterior of the appliance to itsinterior and improve the aesthetic appearance of the appliance. Second,because the speaker mesh element 2800 is formed of a frosted,light-transmissible material, it can act as a diffuse light spreader todiffuse and spread the light emitted from the light interactive element.

FIG. 3D generally illustrates the appearance of the speaker-containingportion of the appliance 2000 once speaker fabric 2806 is attached tothe speaker mesh. The figure also illustrates the manner in which thespeaker mesh can generally diffuse the light provided by the lightinteractive element.

In certain embodiments the main structure of the local appliance 2000will be formed through the coupling of a back cover element cut to thefront chassis element 2100 described above. FIGS. 3E and 3F illustrateaspects of such an exemplary back cover element 2900 with FIG. 3Eillustrating the rear, externally visible, aspects of the back coverelement 2900 and FIG. 3F illustrating the interior side of the backcover element 2900. The back cover element 2900 may be formed fromplastic, polycarbonate, ABS, or a blend of any of the foregoing.

Referring first to FIG. 3E, it will be seen that the exemplary backcover defines several openings 2901 (only two of which are identified inFIGS. 2E and 2F) through which connecting elements, such as screws, maybe passed to connect the back cover 2900 to the front chassis 2100.Screw caps 2902 may be used to cover these openings once the connectionis made.

As further shown in FIGS. 3E and 3F, connecting components 2903 (two ofwhich are labeled in FIG. 3E) may be included in the back cover topermit connection of various mounts to the appliance. For example,connecting elements in the form of screw or bolt-receiving receptaclesmay be located within the back cover to enable mounting structurescomplying with the VESA standard for mounting devices.

In the illustrated example the back cover defines a flat, generallyrearwardly-extending portion 2910 that can form a base that enables theappliance to stand upright on a flat surface. As shown in the figure,this extending portion extends from a first point on the back cover toan extended point such that it defines generally triangularly-shapedside sections 2912 (only one of which is labeled in FIG. 3E. As bestshown in FIG. 3F, this rearwardly extending portion defines an interiorcavity 2914 in which larger components of the appliance (such as thebattery) may be positioned.

As further shown in FIG. 3F, the back cover defines a number of thermalvents, including side thermal vents 2916 located near the bottom of theback cover 2900 on both sides of the triangular sections and upper topvents 2918 located along the top of the back cover. These thermal ventsare arranged such that they can permit hot air and gasses to flow fromthe interior of the appliance, through the top thermal vents 2918, tothe ambient environment. These vents also permit the flow of generallycooler ambient air into the interior of the local appliance 2000(through the side vents 2916) upon over and across the internal elementsin the appliance (so as to cool them) and then out of the appliancethrough the top vents 2918.

As shown in FIG. 3F anti-dust fabric elements 2920 can be coupled to theinterior portion of the back-cover across the thermal vents to limit thepassage of dust or other materials from the outside of the appliance toits interior.

In the example of FIGS. 3E and F, additional features may be formed inthe rear-portion of the back cover, such as a Kensington security lockfeature 2930.

Another additional feature may be a recessed area 2940 for supporting anappliance ON/OFF button. The use of a recessed area, located at the rearportion of the appliance (when the appliance is facing an interviewee)is beneficial because it prevents inadvertent activation/deactivation ofthe appliance and because the recessed, rear-located ON/OFF button makesit generally unobservable and inaccessible to an interviewee such thatinterviewees will not likely be motivated or inspired to depress thebutton (or switch).

One additional feature that may be formed into the back cover is asoft-locking recessed connection port 2960 such as the port shown inFIG. 3E. As reflected in the figure, the soft-locking recessedconnection port defines a recessed area that provides for openingsthrough which one or more connections can be made to the componentspositioned within the interior of the appliance. Such connections caninclude, for example, power connections, audio/visual connections (e.g.,HDMI), and data connections (such as a USB connection).

Although not separately illustrated in FIGS. 3E or 3F, an anti-skid padmay be affixed to the bottom of the appliance to inhibit movement whenthe appliance is placed on a flat surface. Additionally, protectivelenses and/or bezels may be placed over various components within theappliance.

FIGS. 3G-1 and 3G-2 illustrate the manner in which the oft lockingrecessed connection port 2960 described above can be used to formsoft-locking port. As shown in these figures a flexible locking port cap2962 may be provided that includes finger elements that engage withopenings in the back cover and shaped openings suitable for receivingcables connected to various connecting elements. As shown in thefigures, the flexible locking port cap is movable form a first, unlockedposition, as shown in FIG. 3G-1 to a second, locked position, as shownin FIG. 3G-2 . In the unlocked position access to the connecting portswithin the recess is readily provided such that connections can be madeand any connecting cables can be placed within the shaped openings. Asshown in FIG. 3G-2 , the flexible locking port 2962 can then be moved toa locked position where it is held—via compression—within the connectionrecess such that the connections are secured and the connecting elementsare protected from the ambient conditions. In one example, the flexiblelocking port is formed from compressible thermoplastic polyurethane.

FIG. 3H illustrates an example where a flexible locking port cap (withthree shaped openings as opposed to the two shown in FIGS. 3G-1 and 3G-2) is shown in its soft locked position.

FIGS. 4A-4C illustrate various components and elements positioned withinthe interior space defined by the chassis 2100 and the back cover 2900.

Referring first to FIGS. 4A-4B, the main circuit boards and electricalcomponents of the local appliance 2000 are illustrated. As will be seenin the exemplary system several main circuit boards are providedincluding: (i) a main carrier board 3010; (ii) an embedded display port(eDP) bridge board 3020; (iii) an input/output docking board 3030; and(iv) a LED support board 3130.

In the illustrated example, the main carrier board 3010 includes asystem-on-module element which can take the form of a QualcommSnapdragon based SOM. A communication chipset (which may utilize anysuitable communication protocol or standard such as 4G or 5G) may belocated on the carrier board and a connection may be made between thecarrier board 3010 and an antenna 3112 to enable wirelesscommunications. A system battery 3113 is positioned between the maincarried board 3110 and the input/output docking board 3130 and betweenthe main carrier board 3110 and the LED support board 3130

The main carrier board may be connected to other components in thesystem via any suitable connecting structures including hardconnections, ribbon connectors, wires, etc. in the example of FIGS.4A-4B, for example, wires 3114 are used to connect components on themain carrier board 3110 to the speakers while one or more ribbonconnectors are used to connect the main carrier board 3110 to the eDPbridge board 3020.

In the examples of FIG. 4A-4B, the use of the eDP bridge board 3020 andribbon connectors coupling the bridge board 3020 to the main carrierboard 3110 form a easily modifiable flexible PCB component contains theconnections necessary to both receive signals form the main carrierboard 3110 corresponding to the images to be displayed on the displayscreen 2200 and to provide the signals necessary to drive the specificdisplay screen component used in the system. This use of the flex-PCBcomponent provides a high degree of flexibility that permits the maincircuit board components of the illustrated system to be used, with noor minimal modification, with different display elements. For example,for display screens that can receive the signals generated natively bythe display processing board or the I/O board, the flex-PCB element maycontain only connectors that pass the signals from the displayprocessing board (or I/O board) to the display. If an alternate displayis used (e.g., one from a different manufacturer, or having a differentscreen size) then the flex-PCB element can be modified to include theconnections and/or processing components necessary to convert thesignals provided by the display processing board (or I/O board) intosignals suitable for driving the alternative display. Thus, the flex-PCBelement permits the basic core components of the local appliance 2000 tobe used with a variety of different displays and/or different displaysizes (e.g., 10 inch, 12 inch, 14 inch).

As best shown in FIG. 4B, in the exemplary embodiment a heatsink 3116 iscoupled to the rear portion of the main carrier board to dissipate heatgenerated by the components on the board.

Referring back to FIG. 4A, it will be noted that several of thecomponents that generate meaningful heat during their operation(specifically the infra-red illuminators 2410, 2410, the speakers 2610,2620, the eye-tracker 2500, and the LED light interactive device 2700)are all located towards the bottom of the appliance) such that heatgenerated from such components will tend to rise within the device. Inparticular, it will be appreciated from FIG. 4A and 4B that—with theexception of the components on the carrier board—the heat producingelements identified above (specifically the speakers, IR illuminators,and eye tracker) are all located within the lower ⅓ of the appliance2000. Considering FIGS. 4A and 4B in connection with FIGS. 3E and 3F, itwill also be appreciated that the lower thermal vents 2916 are alsolocated lower ⅓ of the appliance 2000. This arrangement of severalheat-producing components and the lower thermal vents in the lower ⅓ ofthe appliance will result in the establishment of an airflow path, suchthat ambient air will be pulled into the interior of the appliancethrough the lower side thermal vents as the air heated by the identifiedheat-producing components rises and up and out the top thermal vents2918 This created flow path will cause air to pass over substantiallyall of the major components of the system including, specifically, theheat sink 3116.

As reflected in FIG. 4B, the heat sink 3116 is arranged such that, whenthe appliance is resting on a surface, it has a plurality of finsextending, in a substantially horizontal direction. This arraignmentthus results in a structure where the airflow created through thearrangement of the arrangement of the thermal vents 216 and 218 willresult in an air flow that primarily directly impacts only the lowershorizontal fin. As such, the lower fin can provide a form of a “thermalbuffer” between the airflow rising through the interior of the applianceand the other horizontal fins of the heat sink 3116 which will notencounter such a direct flow.

Additional details concerning certain thermal management techniques arereflected in FIG. 4C.

Turning to FIG. 4C a side cutaway view of an exemplary local appliancedevice 2000 is shown. As depicted, the device includes a back coverelement 2900 that defines a plurality of top thermal vents 2918. Theback cover element 2900 is coupled to the front chassis element 2100,portions of which are labeled in FIG. 4C.

In FIG. 4C, an LCD display element 2700 is positioned with a recess ofthe front chassis and a protective cover 2201 is positioned over the LCDdisplay. A main carrier board 3010 is positioned within an interiorspace defined by the combination of the front chassis element 3100 andthe back cover element 2900. In the example of FIG. 4C a sub-boardcontaining a system on module board 3011 is coupled to the main carrierboard 3010. In one exemplary embodiment, the SOM board 3011 can includea suitable SOM system, such as the Qualcomm® SnapDragon™ SOM. In theexemplary embodiment, the SOM board 3011 incudes a processor 3012.

As depicted in FIG. 4C, a heat sink 3116 is arranged such that it restssubstantially over the processor 3012 on the SOM board 3011. Althoughnot specifically illustrated in FIG. 4C, thermal past may be usedbetween the heat sink 3116 and the components on the SOM board 30111that it overlies (e.g., the processor 3012) to promote thermalconduction between the components and the heat sink. It will beappreciated that thermal paste can be used in any regions of theillustrated system wherein thermal conductivity is desired to beestablished or maintained. In addition to—or as an alternative to—theuse of thermal past, conductive tape may be used to promote and maintainthermal contact.

In the example of FIG. 4C, the heat sink 3116 is positionedsubstantially in the top ⅓ of the appliance 2000, when the appliance isresting on a generally flat surface. The heat sink 3116 is alsopositioned such that its centerline (across the width of the heat sink)is substantially aligned with the centerline of the overall appliance(with respect to the width of the appliance. It will be noted that theheat sink 3116 in FIG. 4C defines a number of horizontally extendingribs and that the horizontal length of the ribs varies form the top tothe bottom of the heat sink. Thus, for example, the heat sink 3116defines a first rib 3117A having a horizontal length that is less thanthe horizontal length of a second rib 3117B. Still further, in theexample, the heat sink 3116 defines a third rib 3117C that has ahorizontal length that is greater than that of the second rib 3117B.

In the illustrated structure of FIG. 4C thermally conductive gap pad3118 (e.g., a pad formed from MCS30 material) is positioned in a spacebetween a portion of the front chassis 2100 that supports the LCD andthe main carrier board 3110. The use of such a gap pad helps promotethermal transfer between such components.

As shown in FIG. 4C a battery 3113 is positioned within the interiorspace defined by the front chassis element 2100 and the back coverelement 2900 at a location below the heat sink 3116 and the main carrierboard 3010.

It will be appreciated that, in the illustrated example described above,the thermal management features discussed herein provide a significantdegree of cooling such that significant heat-producing components can beused without the need of a powered air-moving device, such as a fan.Alternate embodiments are envisioned, however, where active cooling(e.g., a fan) can be used without departing from the teachings of thepresent disclosure.

In certain applications, for privacy and other reasons, it may bedesirable for the local appliance to not retain—or even possess at anytime—information or data that can be readily used to unique identify thespecific human individual being interviewed. In other words, in certainapplications it may be beneficial—to the extent possible—to ensure thatthe discernment process is effectively anonymous in terms of associatingthe received data and the resultant discernment analysis with a specificunique individual. Various processes and methods may be built into thelocal appliance to achieve this goal.

In one exemplary embodiment, a unique identifier may be generated foreach interviewee that is not directly usable to determine the identityof the human being interviewed.

In this embodiment, the system (ether within a given human interfacesystem or in the discernment server system) may—at the inception of aninterview—generate a random identifier for that interviewee, provide therandom identifier to a user of the system (such that the user couldassociate the interview with a specific human individual) and thenprovide a discernment report that provides the analytical output forthat specific random identifier. The human user of the system, then,could use the report to make a determination with respect to the humanindividual (e.g., whether to allow them to pass through security).

The code or token used in the processes described above need not begenerated randomly as long as the code or token itself cannot readily beused to identify a specific human individual. As such, the code or tokencould be simply a sequential number associated with a specific humaninterface system, a string based on non-identity data (e.g., a stringidentifying a location, a human interface system station and, thespecific date and time the interview was initiated).

Still further the code or token used by a discernment system constructedin accordance with the teachings of the present disclosure may begenerated externally (through a process where the external system thatcould retain an association of the token with a specific humanindividual), such that there is an external system that may associatethe code or token with a specific human identity, but where thediscernment system does not have data that readily enables such anassociation. For example, if a discernment system constructed accordingto teachings of this disclosure is used for airport access control, theairport may have a system that generates—for each user—a airport travelID that is associated with a particular user or a particular flightreservation. That travel ID could then be provided by the airport (orused by the airport to generate a further code or token) that is thenprovided to the discernment system. In such a situation, the dataprovided to, and processed by, the discernment system could not be useddirectly to identify a unique human identity associated with a giveninterview. The airport system, however, which would have data linkingthe code or token provided to the discernment system to a given uniqueindividual.

The code or token described herein may take several forms. For example,it could take the form of a paper printout that is scanned or read. Itcould take the form of a physical token provided to a potential humaninterviewee. It could further take the form of information reflected ina visual display, such as a QR code on a smartphone display.

One limitation of the approach described above (where anonymous codes ortokens are used to identify each interviewee) is that the data capturedand analyzed for purposes of the interview assessment could potentiallybe analyzed to determine the specific human identity of the interviewee.For example, even if an anonymous code or token is used to identify aninterviewee, if a malicious actor were to obtain video associated withthe interview or an audio of the interview, they may be able to analyzeand/or process the captured audio or video data to determine the humanidentity of the individual associated with the interview (e.g., throughfacial or voice recognition processes). To protect against suchpossibilities, embodiments are envisioned wherein some or all of theindividual human interface systems within the discernment system mayperform a process to anonymize certain sensor data as it is received (orprior to its transmission to the discernment server system).

For example, in one of many envisioned embodiments, a software,hardware, or combination software/hardware process may be used totransform certain individual sensor data as it is received into datathat cannot be readily used to reconstruct the originally receivedsignal and/or determine the unique identity of the human associated withthe received signal. In accordance with such embodiments a receivedsignal (such as a video or audio signal) may be passed throughstatistical processing and/or an alternative process (e.g., anobfuscation process) to generate a signal that contains or identifiesthe informational content necessary for the desired discernment analysis(e.g., content that enables determination of the overall body posture ofthe interviewee but that does not enable the use of facial recognitionsoftware).

In alternate embodiments, signals received at the human interface systemby sensors may not be modified directly, but rather may be aggregatedwith other received signals to form an aggregated data package that maythen be processed in such a way (e.g., through statistical manipulationor passage through a one-way process) to generate a data package thatmay be adequately used for discernment analysis but cannot be readilyused to identify the specific human individual that resulted in thegeneration of the data.

In certain embodiments, only some of the signals received by the humaninterface system may be anonymized. Signals not readily associated witha particular human identity, such as weight, temperature, heart rate,need not necessarily be anonymized. Signals more readily associated witha specific individual, such as the audio or visual signals, however, maybe processed to generate anonymous data using one or more of theprocesses described herein.

As a still further embodiment, applications may be envisioned wherein itmay be desirable to anonymize data as described herein throughout thesystem, but still retain the ability—in very limited instances andsubject to very tight user access controls—to associate signalsassociated with a given interview with a specific human interviewee. Insuch instances one or more limited access hardware (or software orbiometric or a combination) keys or tokens may be generated that arecapable of reversing the anonymization process and recapturing eitherthe original interview data and/or other data that would allow anauthorized user to associate a given interview with a unique humanidentity. In such embodiments, the ability to anonymize data generallywould be limited by the security measures associated with the describedkeys or tokens. Such systems may be operated with mandatory accesscontrols (MACs) and/or discretionary access controls (DACs) such as areused in trusted computing environments.

In the embodiments discussed herein, the time relationships between oneor more of the various attributes detected by the appliance 2000 andeach other and/or the time relationships between one or more of thevarious attributes detected by the appliance 2000 the stimuli providedto the human interviewee can be used for assessing the state of thehuman interviewee. For example, a facial expression or a change in vocalpitch time-associated with the provision of a specific stimuli may beindicative of one state, while the same detected attribute may bemeaningless if not time associated with such stimuli. As anotherexample, the time coincidence of one attribute, such as a specificfacial brow position and another attribute, such as a change in pupilsize, may be indicative of deception or a concerned state, while theindividual occurrence of either attribute may not. To permit thedetermination of time relationships, the disclosed appliance 2000 caninclude features and processes for time synchronization.

In the disclosed exemplary appliance, various components operate atdifferent speeds. Thus, for example, the eye tracker could be updateddata at a 120 Hz frequency, while the received audio data could beupdated at a 32 KHz frequency, while the received video data could beupdated at a 60 Hz rate, while the interview script is rendered at a 60Hz rate as well. To permit time synchronization of the provided stimuliand the received sensor data, the disclosed system can implement aprocess such as the one depicted in FIG. 5 .

In such an embodiment, the appliance 2000 is used to generate multipledata streams during each interview. Each data stream can comprise datathat is continuously streamed from the appliance to the cloud-baseddiscernment system 1100. Such streams can include, for example, an audiodata stream, a data stream reflecting the received video data, and adata stream reflecting eye tracking data. Each of the data streams willbe time stamped such that each data item (or each interval of dataitems) will be associated with a specific timestamp.

The timestamps provided with each data stream may be either associatedwith a specific given time at the appliance 2000 (e.g., a local timeindictor), a generally running timestamp for the specific appliance, ora relative timestamp associated with a time interval that begins whenthe first interview question is posed to the interview. The appliancemay further provide data associating timestamps with the start and stoppoints for each question posed to the interview and the start and stoptimes of each answer provided by the interviewee. In this manner, thecloud-based discernment system will be provided with the data sufficientpermit it to associate the received streaming signals from the appliancewith the questions posed during the interview. The cloud-baseddiscernment system can then corollate and synchronize the receivedstreaming signals with the various questions and answers and use thesame to analyze the received data.

FIG. 5 illustrates one exemplary embodiment for providing thetimestamped data streams and the question and answer meta-data discussedabove. Referring to the figure, the illustrated system includes atimestamp generator that generates time stamps in microseconds. Asdescribed above, the timestamp generator can be a continuously runninggenerator, one that tracks local time, and/or one that generates timestamps beginning at an initial time (e.g., start of interview). Asreflected in FIG. 5 , the generated timestamps are then associated withvideo frames, audio buffered frames, or eye tracker buffered frames(which in the illustrated example are provided as test frames) and thetime stamped video, audio and eye-tracker data is then streamed to thecloud-based discernment system through use of a JNI interface and a RTPpusher. Thus, through use of the system described above, the exemplaryappliance provides independent timestamped-data streams to thecloud-based discernment system.

The specific form of the streaming data provided by the local appliance2000 may vary depending on the type of data and the particularapplication to which the local appliance is applied. For example, forthe eye tracking data, the streamed data may comprise frames, whereineach frame includes a data set associated with a given timestamp thatprovides: an indication of whether both the left and the right eyelocations were detected, the detected locations (e.g., in X, Ycoordinates) for each detected eye, a weighted average between the twoeyes (e.g., weighted X and weighted Y locations); a measurementassociated with any detected pupil measurements; and/or a detecteddistance measurement. The cloud-based discernment system can thenprocess the received data to either ignore data deemed invalid, processreceived data to smooth invalid or aberrational data, and/or generateerror information in the event that the received data suggests an errorin the system and/or inadequate received data.

In the example of FIG. 5 , in addition to providing the timestampedstreaming data, the illustrated local appliance also provides—thoroughanother communication channel—timestamped data associated with theparticular questions posed to the interviewee and the answers received.This is done via a separate API interface that provides the cloud-baseddiscernment system, either on a regular basis during an interview or atthe conclusion of an interview, information associating the start andstop of each question posed to the interviewee during the interview andthe start and stop of each received answer. Such data, in theillustrated example, is provided over an internet-based encryptedcommunication channel. FIG. 6 illustrates an example of the type of datathat is provided for the questions and answers. As reflected in theexample, the data provided over the API interface associated timestampswith the start and stop of each question and answer.

It will be appreciated that, in an actual implementation, many localappliances will be operating to provide interviews at the same time. Assuch, the cloud-based discernment system will be receiving multiple datastream feeds and multiple question and answer feeds. However, since eachfeed will be associated with a particular unique interview ID, andbecause all of the provided data will be time-stamped, the cloud-basedserver will have the ability to synchronize the questions, answers, andreceived video/audio/eye-tracker data for all interviews such that itcan provide assessments and reports for a large number of concurrentlyoccurring interviews. This ability is important for successful operationof a large, scalable, distributed discernment system

To control the flow of audio data from the local appliances to thecloud-based processing system, the exemplary embodiment disclosed hereinincludes apparatus and processes for determining: (1) whether receivedaudio data corresponds to human voice activity and (2) whether receivedaudio data corresponding to voice activity represents a completeresponse to provided stimuli (i.e., whether received data correspondingto voice activity corresponds to a complete verbal response to a posedquestions). These apparatus and process are important in the context ofa distributed system, such as the one disclosed herein, because theytend to ensure that only voice activity audio data is transmitted to thecloud processing system (thus avoiding bandwidth burdens that would beassociated with transmitting non-voice activity data, such as backgroundnoise, non-vocal sounds (e.g., dog barking, etc.)). These apparatus andprocesses are also significant because they ensure that complete verbalresponses to provided stimuli are provided to the cloud processingsystem (thus promoting overall system accuracy) and they ensure that thesystem does not move from one provided stimuli to the next (e.g., formone verbal question to the next question) until a complete response tothe prior stimuli is received.

FIG. 7A illustrates on exemplary system 7000 that may be included andimplemented within the apparatus 2000. As reflected in the figure, theexemplary system includes: a microphone 7100; a vocal activity detector7200; an end-of-speech detector 7300; and some processing logic 7400 and7450 for advancing to the next stimuli (or to the end of the interview)or to another activity once it is determined that a complete response tothe prior question was received.

In the example of 7A, the microphone 7100 may take the form of amono-microphone sampling generating digital sampled audio signals at arate of 16 KHz with a sampling rate of approximately 16 bits/second. Thevoice activity detector 7200, the end of speech detector 7300 and theprocessing logic 7400 and 7450 may take the form of dedicated processingelement (i.e., a dedicated circuit or a FPGA), a programmed processorthat also performs other functions, or a combination of a dedicated andprogrammed element.

In the described embodiment, the sampled audio data from the microphonewill be packaged into data frames, each of which will correspond to aparticular time period. In one embodiment, each data fame willcorrespond to a time sampling period of 30 milliseconds.

As reflected in FIG. 7A, each sampled audio frame is processed by thevoice activity detector 7200 to determine whether the sampled audiowithin the frame includes data corresponding to human voice activity.This processing can be accomplished in a variety of different ways. Inaccordance with one embodiment, this processing is accomplished bytaking the sampled audio signal, splitting it into frequency bandscorresponding to frequencies associated with human voice activity andthen determining: (i) whether the total power associated with thehuman-associated frequencies is above a minimum threshold (which wouldsuggest human speech directed to the microphone 7100) and, if so, (ii)whether the data associated with the various frequency bands is likelyto be associated with human voice activity.

Once the received data frame is processed by the voice activity detector7200, the received audio frame can be passed to the end-of-speechdetector 7300 along with an indication of whether the received audioframe is associated with speech or no speech. The end of speech detector7300 can then process the received data to determine whether anend-of-speech event has occurred (e.g., whether a complete verbalresponse to prior stimuli has been received).

FIG. 7B shows one process that may be implemented by the end of speechdetector A300 to determine whether an end of speech event has occurred.In the illustrated example, the end of answer process is initiated eachtime a new question is posed to an interview during an interview.

Turning to the figure, at a step 7701, the end-of-speech detector 7300will receive the audio data and will initially wait until the timeinterval elapsed since the last processing step is equal to or greaterthan the frame duration (“fd”) such that the system has a full frame ofaudio data. In the depicted example, the frame duration is set at 30msec.

Once a frame of audio data has been received, the system will move tostep 7702 and analyze the data from the vocal activity detector todetermine whether the received audio frame corresponds to human speech.If the received frame DOES correspond to human speech the system willthen update the speech duration interval (“sd” in the example) by addingto the previous interval the time that has elapsed since the lastprocessing step occurred. This is done at step 7704.

The system will then proceed to step 7706 where it will determinewhether the previously received and running detected human vocalactivity period is less than a given threshold period (in the example,twice the frame duration or 60 ms). This step is used to determinewhether the receive human activity is—in fact—speech (which willtypically have a duration in excess of the given threshold period) orsome other form of human generated sound. If it is determined that step7706 that the previously received running detected human vocal activityIS above the given threshold period, then the system will conclude thatspeech is found and set the no speech duration to zero and set a speechfound indicator to indicate that speech has been detected at step 7708.The system will then go back to step 7701 and await the receipt of asubsequent audio frame.

If it is determined in step 7706 that the previously received runningdetected human vocal activity IS NOT above the given threshold period,then the system will conclude that it is not yet clear whether humanspeech has been detected and go back to step 7701 and await the receiptof a subsequent audio frame.

If the system determines at step 7702 that NO SPEECH is detected it willthen proceed to a step 7722 where a no speech duration counter (nsd) isincremented by adding the interval that elapsed since the last audioframe was processed to the no speech detected counter value. The systemwill then proceed to a step 7724 where it will determine whether the nospeech detection interval is greater than a set noise threshold (nsth).In the illustrated example, the noise threshold (nsth) is set at sixtimes the frame duration (or 300 msec). The purpose of the step 7724 isto determine whether the frame reflecting a lack of human voice activityis indicative of a genuine lack of human speech as opposed to a gap inhuman vocal activity, such as a gap between spoken words, sentences, orbreaths that reflect a lack of instantaneous vocal activity but not alack of on-going human speech.

If it is determined at step 7724 that the ongoing cumulative period ofno human vocal activity has exceeded the noise threshold, the systemwill conclude that there is no ongoing human speech and re-set thespeech duration (sd) (e.g., to zero) at step 7726.

If it is determined at step 7724 ongoing cumulative period of no humanvocal activity is less than the noise threshold, the system will notreset the speech duration variable.

Under both circumstances, the system will then proceed to step 7728.

At step 7728, the system will determine whether the speech foundvariable was previously set or not. This step will reflect whether thecurrent period of no human vocal activity follows a period where speechhad been detected (such that it may reflect the end of an answer—asopposed to the lack of any answer).

If the system determines that the present period does follow a periodwhere speech had been detected (e.g., one where “Speech Found” had beenset) the system will then proceed to step 7730 where it will determinewhether the current cumulative period of no speech detected (nsd)exceeds the period set for the end of speech threshold (esd). In theexemplary embodiment under discussion, the end of speech threshold isset at 1.5 seconds of continuous no speech. If it is determined that theend of speech threshold has been exceeded (after an interval of detectedspeech) the system will determine that the end of the answer has beenprovided and proceed to step 7732. If not, the system will return tostep 7701 and await the receipt of the next frame of audio data.

If the system determines at step 7728 that the current period of nospeech detected does NOT follow a period where speech had been detected(which could exist where a question has been posed but no answer hadbeen received) the system will then proceed to step 7734 where it willdetermine whether the current period of no speech exceeds both the endof speech threshold (esd) plus a grace period. In the illustratedexample, the grace period is set at 0.5 seconds. The purpose of step7734 is to provide the interview with an amount of time in excess of theend of speech threshold to begin answering a question. This “graceperiod” is required because it will take the interviewee some amount oftime to process any provided questions and begin answering. If thesystem determines at step 7734 that the combined period associated withthe end of speech threshold and the grace period has NOT been exceeded,the system will return to step 7701 and await the next frame. If thesystem determines that the combined period exceeds the end of speechthreshold plus the grace period, it will proceed to step 7736 where itwill determine that no response to the presented question was provided,set a no response indicator, and proceed to step 7732 where the systemwill conclude that the end of the answer to the posed question (in thiscase the answer being the lack of any verbal response).

At step 7732, the system can then proceed to present the next questionto the interviewee, if any interview questions remain to be asked, or toan end of interview process of no questions remain.

In the examples of 7A and 7B, the various values used for the end ofanswer assessment were fixed. Alternat embodiments are envisionedwherein some or all of the assessment parameters are varied. Forexample, is some embodiments, the maximum no speech threshold (“esthr”)used to detect an end of speech event could vary depending on thespecific stimuli presented to the interviewee. Thus, for example, themaximum no speech threshold could have one value for posed questionsthat require minimal mental processing (or thinking) on the part of theinterviewee (e.g., 3 seconds) and a different vale (e.g., 6 seconds) forquestions or stimuli requiring additional processing. Such variationscould be associated with different stimuli types (e.g., no thinking,minimal thinking, intense thinking) in which case the stimuli dataprovided by the cloud processing system for a given stimuli couldindicate the stimuli type. Additionally, or alternatively, some or allof the provided stimuli data could provide specific end-of-speechdetection parameters for those stimuli.

In addition to varying the end of speech detection process in responseto the type of provided stimuli, still further alternate embodiments areenvisioned wherein the end of speech detection parameters are variedbased on known conditions associated with the interviewee. For example,certain of the processing parameters could be varied based on the age ofthe interviewee with interviewees younger than 20 being processed usingone set of parameters, interviewees between 20 and 60 years of age withanother set of parameters, and interviewees over 60 using yet anotherset of parameters.

In the exemplary embodiment only audio data is used to assess whether aninterviewee has provided no response or has reached the end of aresponse to a question posed in an interview. Alternative embodimentsare envisioned wherein additional data feeds are used to assess whetheran interviewee has provided a response or has reached the end of ananswer. For example, in one alternative embodiment visual camera datacan be used to assess whether, over a given audio frame, the intervieweeis exhibiting mouth movement. Such an assessment can be made bydetermining whether, over a given audio frame determined to beassociated with human speech, the interviewees mouth is moving. If it isdetermined that, over a sufficient number of consecutive audio framesover which human vocal activity is detected, there is no correspondinginterviewee mouth movement, a no speech (or no response or end ofanswer) determination can be made.

Note that the use of visual data in connection with the no response/endof answer assessment can also be used to determine whether identifiedhuman vocal activity corresponds to vocal activity of the interviewee.Such uses can be beneficial in environments where the interview is beingconducted in an environment wherein human vocal activity other thanactivity from the interviewee may be detected by the local applianceused for conducting the interview. For example, in an airport screeningsituation, an interview of one interviewee may be conducted at the sametime as other interviews are being conducted in a nearby location withother devices and interviews and/or other ambient human vocal activitymay be detected by the microphone(s) within the local applianceconducting the interview. In such situations, use of visual data

FIG. 7C illustrates an exemplary embodiment wherein both audio data andvisual data are processed to detect an end of answer/end of speechcondition and/or a no response condition.

Turning to FIG. 7C, elements and steps having reference numbers used inconnection with FIG. 7A or 7B have the same functionality. As reflectedin FIG. 7C, the process operates in a manner similar to that describedabove in connection with FIG. 7B except that a determination is made, insteps 7745 or 7744, as to whether a speaking expression has beendetected from processing the visual data. The existence of a speakingexpression, e.g., a detection of mouth or lip movement, provides anindication that the interview is about to begin speaking, is stillspeaking and/or is moving their mouth in a manner associated withspeaking.

In the example, of FIG. 7C, the detection of a speaking expression, ineither of steps 7744 or 7745, will result in the incrementing of aspeaking grace counter in one of step 7746 or 7747. The system will thenuse a period associated with the sum of the end of speech period (“esd”)plus the running speaking grace period (“spegrace”) at step 7743 todetermine whether an end of answer condition exists (for situationswhere speech was previously detected) or the sum of the esd period, plusthe spegraceperiod, plus the beginning answer grace period (“grace”) todetermine whether a no response condition exists at step 7742.

Note that, in the example of FIG. 7C, the speaking grace counter isreset (or set to 0) at the beginning of the process and/or in step 7740(which follows a detection of speech).

The disclosed distributed system allows for the generation of reportsthat may be defined and accessed by a limited set of administrators.

For example, in one embodiment, the system 2000 can provide a reportsite that is accessible only to specifically authorized to access thesite. The report site may contain reports, or data from which reportscan be generated, associated with interviews conducted for that specificadministrator. Because the data associated with each interview wereprovided to the could-based discernment system 1100 on an anonymousbasis, the reports will typically be identified in the Report Site byone or more of: (a) the date, time, location and/or specificallyassociated interview appliance associated with the interview; (b) aspecific interview identifier (e.g., custom generated interview code)and/or (c) other potentially identifying information associated withsome or all of the interviews.

The interview reports generated by the disclosed system can take avariety of different formats and can be accessed in a variety ofdifferent ways.

FIG. 8A illustrates an exemplary report interface.

Referring to FIG. 8A, a report interface is illustrated that may beaccessed by a given Administrator through a Dedicated Tenant Portal, aprotected virtual machine running on a shared device, or hosted site.Note that the report interface, and the reports data, may—in someembodiments—be accessible only by tenant representatives such that theywould not be maintained or stored within the could-based discernmentsystem.

Referring to FIG. 8A, it will be seen that the reports can be associatedwith interview IDs, the name of a specific interview (so that similarinterviews can be grouped) and/or a specific interviewee ID. The reportscan also be associated with an interview date and/or an indication ofthe type of issue or issues addressed during the interview (e.g.,initial screening, security, theft detection, etc.).

In the example of FIG. 8A, color-coded identifiers are provided for eachinterview to indicate whether the particular interview associated withthe identifier poses any significant risk factors. The nature andthreshold for the provision of such identifiers can be set such that agiven identifier (e.g., RED for heightened risk) can be provided when areport meets a first set of criterial a further identified (e.g., GREEN)can be provided when a second set of criterial is met and otheridentifiers (e.g., YELLOW) can be provided when yet other criteria setsare met.

A view report tab can be provided for each interview.

FIG. 8B illustrates an example form of an interview report.

Referring to FIG. 8B, the depicted exemplary interview Report provides arisk assessment for each question posed during the interview both interms of a general qualitative assessment (Normal, High Risk, ModerateRisk) and a graphical numerical assessment. To highlight HIGH riskquestions, the report may include color highlights (or other highlights)that emphasis questions for which HIGH risk responses.

In addition to the general qualitative assessment, the interview Reportmay provide a quantitative assessment of each response to an interviewquestion in the form of a Risk Percentage. Such an assessment may takethe form of a numerical value where the numerical value is mappedagainst a threshold reflecting a normal risk response.

Through the use of the qualitative and quantitative information providedin the interview Report, the reviewer of the report can determinewhether any further actions are required or appropriate.

It will be appreciated that the administrative functions enabled by thedisclosed system can be provided in a variety of different ways fordifferent users of the system. For example, the nature of the disclosedsystem allows the functions provided by the system to be provided bythree different physical or logical function systems or layers: (1) anedge device layer or system that provides the interface between thesystem and the interviewees; (2) a discernment detection layer or systemand (3) an administrative layer or system.

In the disclosed example, the edge detection functionality is providedby the local appliances 2000 and the discernment detection is providedby the cloud-based discernment detection server 1100. In that examplethe administrative functionality can be provided by one or more of theadministrative interfaces 1300 operating in isolation, in cooperatingwith other interfaces 13000 and/or or in conjunction with aspects of thecould-based discernment system 1100.

For purposes of the following discussion, a “tenant” is intended torefer to an organization or entity that is using the discloseddiscernment system 1000 to arrange and conduct interviews and to receiveand review reports related to those interviews. A tenant may be, forexample, an airline, a boarded security service, an employer, an accesscontrol group (e.g., a group controlling entry/exit access to/form asporting event), or any other organization desiring use of thediscernment detection system.

In accordance with one exemplary embodiment the data and activitiesassociated with multiple tenants can be supports by the discernmentdetection server 1100 (or services) such that the operator of theservers 1100 supports and provides the administrative functionality tothe various tenants. In such an embodiment, however, no specificresources would be shared among different tenants and each tenant wouldaccess and operate the system within an isolated private (or virtualprivate) network. In this embodiment, it will be appreciated that eachtenant will operate using the services provided by the cloud-baseddiscernment servers 1100 but on a different, and securely isolated,private network.

In other embodiments, one or more tenants may require that their tenantadministrative functions be provided at locations — or using equipment —completely under their control. Such embodiments may be required, forexample, where a local boarder security organization or a localcorporate entity wishes to maintain complete control over alladministrative data. As noted above, in certain implantations of thesystem under discussion, all data received and transmitted by the localappliances 1200 to the could-based discernment detection servers 1100 isanonymized such that only certain tenant-accessible resources could haveaccess to data that would allow one to associate an interview—and anyinterview associated data or reports—with a given human individual. Toenable tenants desiring complete control over such—and other data—alladministrative functionality that could be used to uncover or infercertain sensitive information (such as the human identity associatedwith any interview, interview data or report. In such embodiments, mostof the administrative functions will be provided by on-siteadministrative functionality. In such examples, the local appliancedevices 1200 will be used to unanimously collect and transfer data tothe cloud-based discernment detection servers 1100 and the cloud-baseddiscernment detection servers 1100 will be used to generate reports andtransmit such reports to the various tenant-specific devices providingthe administrative functions. Thus, in these embodiments, thediscernment-detection interviews, analysis, and reporting is all done inan anonymous environment using non-tenant specific hardware andnetworks, while some or all of the administrative functionality thatcould be used to determine or infer the human-specific interviewees, areprovided by tenant-implemented, tenant-controlled hardware and networks.

The Communication Links: The communication links between the humaninterface systems and the discernment server system may take anysuitable form, such as wired connections or wireless connections. Incertain embodiments involving mobile devices, the communication channelsmay include wireless communications with some or all of the humaninterface systems (e.g., through high-speed, high-bandwidth 5Gconnections) coupled with downstream wired connections or furtherwireless connections.

Encryption: As a security measure, all or part of the data used in adiscernment system constructed in accordance with teachings of thisdisclosure may be encrypted both as it is communicated across anycommunication link and as processed within the system. Thus, forexample, the data received by the human interface system may beencrypted, the data transmitted from the human interface system to thediscernment server system may be encrypted, and all reports and/oranalysis generated by the discernment server system may be encrypted.Note that any such encryption could be distinct from — or integratedwith — the anonymous processes discussed previously.

Thus, for example, one could encrypt data that has not be anonymized(such as a non-anonymized video file). While such a file would beencrypted—in the sense that it would not be readily accessed by thosenot authorized to receive and view such data—it would not be anatomizedbecause anyone able to decrypt the data file could then use it toidentify the unique human associated with the file.

In other embodiments one could both anonymize and encrypt data used bythe discernment system either through separate processing steps orthrough an integrated process where input non-anonymous data is bothanonymized and encrypted through a single process step.

As is known to those sufficiently skilled in the art, in addition tousing encryption to protect the data from observation, encryption may beused to authenticate the data. That is to say that a device holding aprivate encryption key may encrypt that data such that it may beverified as being encrypted by that device at a later time.

In the exemplary embodiment disclosed herein, a multi-layered approachcan be implemented to secure the data used by the system, both withrespect to any stored data and/or with respect to the transmission ofdata.

Data in transmission can be secured and protected by using one or moresecure transmission protocols, such as one of the various transportlayer security protocols (e.g., TLS 1.2).

In terms of stored data, the data can be stored in an encrypted formatand access to the stored data can be limited to accessed users throughthe use of conventional security, network security, operating systemcontrols are applied to protect the data from security vulnerabilities.

The Discernment Server System: In one exemplary embodiment thediscernment server system (or systems) will take the form of a server ormultiple servers that communicate with the human interface systems to atleast: (a) provide most or all of the information necessary to providestimuli to the interviewee for an interview; (b) receive detectedinterview data from the human interview systems; (c) process receiveddetected data in light of the provided stimuli to generate furtherstimuli interactions with an interviewee and/or to assess and analyzethe received signals and to provide a report or indication reflectingthe underlying state of the human interview; and/or (d) provide aninterface into the discernment server system that may be used to modifythe system, adjust the nature of one or more interviews, directlycommunicate with a human interface system, monitor an interview in realtime, or request to generate various reports. Other functionality may beenabled by or within the discernment server system.

The precise process by which the discernment server system assesses theunderlying state of a human interviewee may vary significantly. Forexample, in applications where the system is deployed to detectdeception at an airport access point, the discernment server system mayinclude one or more machine learning models—created through the use ofsignificant test data—that correlate certain received signals from thehuman interface system with deception. For example, in applicationswhere an interviewee is asked whether they are transporting certaincontraband, and a question is posed with a visual depiction of thecontraband, the vocal inflection of the interviewee along with anassessment of the interviewee's eye gaze pattern (e.g., are the focusingon or avoiding focusing on the displayed contraband in an unusualmanner) can provide an indication about whether the interview is beingtruthful in their response. Various approaches for detecting the stateof a human interviewee using received sensor data are discussed anddisclosed, for example, in U.S. Patent Application Publication No.2013/0266925.

The physical implementation of the discernment server system may takemany forms. In one embodiment the discernment server system may be acomputer server (or group of servers) dedicated solely to thedistributed discernment system. In other embodiments the discernmentserver system may be implemented virtually in the cloud such that it isnot temporally linked to any specific physical hardware. Hybridapproaches are also envisioned.

Other and further embodiments utilizing one or more aspects of theinventions described above can be devised without departing from thespirit of Applicant's invention. Further, the various methods andembodiments of the methods of manufacture and assembly of the system, aswell as location specifications, can be included in combination witheach other to produce variations of the disclosed methods andembodiments. Discussion of singular elements can include plural elementsand vice-versa.

The order of steps can occur in a variety of sequences unless otherwisespecifically limited. The various steps described herein can be combinedwith other steps, interlineated with the stated steps, and/or split intomultiple steps. Similarly, elements have been described functionally andcan be embodied as separate components or can be combined intocomponents having multiple functions.

The inventions have been described in the context of preferred and otherembodiments and not every embodiment of the invention has beendescribed. Obvious modifications and alterations to the describedembodiments are available to those of ordinary skill in the art. Thedisclosed and undisclosed embodiments are not intended to limit orrestrict the scope or applicability of the invention conceived of by theApplicants, but rather, in conformity with the patent laws, Applicantsintend to protect fully all such modifications and improvements thatcome within the scope or range of equivalent of the following claims.

What is claimed is:
 1. A distributed discernment system comprising: adiscernment server including a processer running a diagnostic programand a communications interface permitting bi-directional communicationsto and from the discernment server; a plurality of human interfacedevices, each human interface device including a speaker for presentingaudible stimuli to a human interviewee, a microphone, a processorrunning a local processing program, and a system interface permittingbi-directional communications between the human interface device and thediscernment server; and a communications network permittingbidirectional communications between the discernment server and each ofthe plurality of human interface devices wherein: the diagnostic programrunning on the discernment server is adapted to generate interviewinstructions to be provided to the plurality of human interface devicesover the communications network, wherein the instructions provided to agiven human interface device cause the human interface device to presentaudio stimuli to a human interviewee in the form of a verbal question;each of the plurality of human interface devices is adapted receiveinterview instructions from the discernment server over thecommunications network and the local processing system is configured, inresponse to such instructions, to: present audible stimuli to a humaninterviewee in the form of the verbal question; receive sensor data fromthe microphone following the presentation of the audible stimuli;process the microphone sensor data to determine whether the microphonesensor data corresponds to a complete human voice response to thepresented verbal question; and if the microphone sensor data isdetermined to be a complete human voice response to the presented verbalquestion, provide data to the discernment server indicating that acomplete response to the presented verbal question has been received;and the discernment server is adapted to receive data from each of aplurality of the human interface devices and analyze such data toprovide an assessment of the state of the human interviewee interactingwith each such human interview device.
 2. The distributed discernmentsystem of claim 1 wherein, each of the plurality of human interfacedevices further comprises an eye tracker, and wherein: the microphonegenerates a timestamped audio data stream; the eye tracker generates atimestamped eye tracking data stream, and wherein each of the pluralityof human interface devices further comprises: means for streaming thetimestamped eye tracking data stream and the timestamped audio datastream to the discernment server in such a manner that the data streamsare associated with an interview ID; means for generating timestampsassociated with the start and stop points of the presentation of theverbal question to a human interviewee interacting with such device;means for generating timestamps associated with the start and stoppoints of the human interviewee's response to the presented verbalquestion; and means for transmitting to the discernment servertimestamped data associated with the start and stop points of thepresented verbal question and timestamped data associated with the startand stop points of the human interviewee's response to the presentedverbal question.
 3. The distributed discernment system of claim 2wherein the timestamped eye tracking data stream and the timestampedaudio data stream are transmitted to the discernment server over a firstcommunication channel and the timestamped data associated with the startand stop points of the presented verbal question and timestamped dataassociated with the start and stop points of the human interviewee'sresponse to the presented verbal question are transmitted to thediscernment server using a second channel, where the first and secondchannels are different communication channels.
 4. The distributeddiscernment system of claim 3 wherein, over at least one given timeperiod: a first one of the plurality of human interface devices isadapted to associate with a first interview ID: (i) the timestamped eyetracking data stream from the first device; (ii) the timestamped audiodata stream from the first device; (iii) the timestamped data from thefirst device associated with the start and stop points of the presentedaudible question; and (iv) timestamped data from the first deviceassociated with the start and stop points of a first human interviewee'sresponse to the presented audible question; and a second one of theplurality of human interface devices is adapted to associate with asecond interview ID: (i) the timestamped eye tracking data stream fromthe second device; (ii) the timestamped audio data stream from thesecond device; (iii) the timestamped data from the second deviceassociated with the start and stop points of the presented audiblequestion; and (iv) timestamped data from the second device associatedwith the start and stop points of a second human interviewee's responseto the presented audible question; and the first interview ID isdifferent from the second interview ID.
 5. The distributed discernmentsystem of claim 1 wherein each of the plurality of human interfacedevices includes on or more processing elements adapted to: package themicrophone sensor data into audio data frames, with each audio dataframe corresponds to the microphone sensor data received over aparticular time period; process each audio data frame to determinewhether the sampled audio within the audio data frame includes datacorresponding to human voice activity; and provide an indication, for atleast a plurality of the audio data frames, whether the data within theaudio data frame corresponds to human voice activity or not.
 6. Thedistributed discernment system of claim 1, wherein each of the pluralityof human interface devices further includes an end of speech detectorfor determining when a human interviewee has provided a completeresponse to a provided first audible question and the local processor isconfigured to present a second stimuli in the form of a second audiblequestion to the interviewee upon the determination that the humaninterviewee has provided a complete response to the first audiblequestion.
 7. The distributed discernment system of claim 6 wherein eachof the plurality of human interface devices determines that a completeresponse to the first stimuli was provided by comparing a detectedperiod of no human speech to a no speech threshold, and wherein the nospeech threshold for a first audible question presented by the interfacedevice varies from the no speech threshold for a second audible questionpresented by the interface device.
 8. A human interface appliance foruse in a distributed discernment system, the human interface appliancecomprising: a front chassis element defining a first generally flatsection and a second section located below the first generally flatsection, where the second section extends generally outwardly from thefirst generally flat section at an angle such that, when the applianceis positioned on a flat surface, the second section will be generallyangled upwards with respect to a human interacting with the appliance;an eye tracker for detecting movement of the eyes of the humaninteracting with the appliance, the eye tracker being associated withthe second section of the front chassis element, and positioned suchthat, when the appliance is positioned on a flat surface, the eyetracker will be generally angled upwards from the horizontal within arange of ten to twenty degrees; a display screen configured to providevisual stimuli to the human interacting with the appliance, the displayscreen being associated with the first generally flat surface of theappliance and positioned such that, when the appliance is positioned ona flat surface, the display screen is above the eye tracker; a digitalcamera for capturing the facial image of the human interacting with theappliance, the digital camera being positioned such that, when theappliance is positioned on a flat surface, the digital camera is abovethe display screen; and a first speaker for providing audible stimuli tothe human interacting with the appliance; a microphone configured toprovide an output signal corresponding to received audio; means forcausing the display and the speaker to present visual and audiblestimuli to a human interacting with the appliance, wherein the audiblestimuli includes the presentment of a first audible question to thehuman; and means coupled to receive the output signal from themicrophone and determine whether the received audio signal correspondsto human voice activity representing a complete response to the firstaudible question.
 9. The human interface appliance of claim 8 whereinthe appliance defines a width and an outer edge, wherein the appliancefurther comprises an ambient light sensor, and wherein: the digitalcamera is positioned at approximately the midpoint of the width of theappliance; and the ambient light sensor is positioned in approximatelythe same horizontal plane as the digital camera at a location closer tothe digital camera than the edge of the appliance.
 10. The humaninterface appliance of claim 9 wherein the microphone is a multi-elementmicrophone and a first element of the microphone is positioned closer tothe digital camera than the edge of the appliance and a second elementof the microphone is positioned closer to the edge of the appliance thanto the digital camera.
 11. The human interface appliance of claim 8wherein: the front chassis further comprises a third section that isangled such that, when the appliance is positioned on a flat surface,the third section will be generally angled downwards with respect to thehuman interacting with the appliance; the human interface furthercomprises a second speaker; and the first and second speakers arepositioned to emit sound through openings in the third section of thechassis.
 12. The human interface appliance of claim 11 furthercomprising a light emitting diode assembly positioned between the firstand second speaker elements.
 13. The human interface appliance of claim8 wherein the means coupled to receive the output signal from themicrophone and determine whether the received audio signal correspondsto human voice activity representing a complete response to the firstaudible question comprises a processor programed to: package the outputfrom the microphone into audio data frames, with each audio data framecorresponds to the sensor data provided by the microphone over aparticular time period; process each audio data frame to determinewhether the sampled audio within the audio data frame includes datacorresponding to human speech; and provide an indication that a completeresponse has been received when it is determined that a period of nodetected human speech has occurred following the detection of a periodof human speech.
 14. The human interface appliance of claim 13 whereinthe processor is programed to detect a period of human speech followingthe presentation of the first audible question, and wherein the periodof no detected human speech used to determine whether a completeresponse has been received to the first audible question is based on theextent to which the response to the first audible question will requiremental processing on the part of the human.
 15. A method of discerningthe state of a human in a distributed discernment system, the methodcomprising the steps of: presenting audible stimuli to the human in theform of a verbal question and generating timestamped data associatedwith the end of the presented audible stimuli; using a microphone tomonitor for a potential response to the presented audible stimuli and togenerate a microphone output signal; streaming the microphone outputsignal to a cloud-based discernment system over a first communicationchannel; packaging the output from the microphone into audio dataframes, with each audio data frame corresponding to the output signalprovided by the microphone over a defined time period; processing eachaudio data frame to determine whether the sampled audio within the audiodata frame includes data corresponding to human speech; determining thatthe human has provided a complete answer to the presented question whena period of no human speech exceeds a no-speech threshold following theend of the verbal question; generating timestamped data corresponding tothe end of the received answer; sending timestamped data to thecloud-based discernment system corresponding to the end of the presentedverbal question and the end of the verbal over a second communicationchannel.
 16. The method of claim 15 wherein the step of processing eachaudio data frame to determine whether the sampled audio within the audiodata frame includes data corresponding to human speech comprises thestep of splitting an audio signal associated with the audio data frameinto frequency bands corresponding to frequencies associated with humanvoice activity, and determining whether the total power associated withthe human-associated frequencies is above a minimum threshold.
 17. Themethod of claim 15 further comprising the step of determining that thehuman has started to provide an answer to the presented verbal questionwhen a period human speech is detected within a first period of timefollowing the end of the presented question and generating timestampeddata corresponding to the start of the received answer; and wherein thestep of sending timestamped data to the cloud-based discernment systemcorresponding to the end of the presented verbal question and the end ofthe presented question, further includes sending timestamped datacorresponding to the start of the received answer.
 18. The method ofclaim 15 wherein the duration of the no-speech threshold is associatedwith the amount of mental processing anticipated to be required of thehuman to respond to the presented question.
 19. The method of claim 15further comprising the steps of: using a camera to monitor facialmovements of the human and to generate a camera output signal; streamingthe camera output signal to a cloud-based discernment system over acommunication channel; and wherein the step of determining that thehuman has provided a complete answer to the presented question includesthe step of processing the camera output signal to determine whether thefacial movements of the human indicate the lack of mouth movement. 20.The method of claim 15 further including the step of processing thestreamed data within the cloud-based discernment system to assess thestate of the interviewed human.